Flux Fourth Release - Second-level live images, Byte open-source Hyper-sd supports Flux, and supports multiple LoRA overlays!

Recently, ByteDance’s open-source Hyper-sd project officially supports FLUX.1-dev. It currently supports 8-step LoRA and 16-step LoRA, and compared to the default 30-step in FLUX.1-dev, the speed has improved by nearly 4 times! Meanwhile, Hyper-sd’s official team also announced that LoRAs with even fewer generation steps will be released soon, and we’re all looking forward to it!

Technical Analysis

The Hyper-SD framework is designed to address the computational overhead in the multi-step reasoning process of diffusion models (DMs). This framework combines the advantages of ODE trajectory preservation and rewriting, maintaining near-lossless performance during the compression steps. Specifically, it uses trajectory segment consistency distillation to perform consistent distillation step by step, retaining the original ODE trajectory within predefined time intervals.

In addition, it also utilizes human feedback learning to improve the model’s performance in low-step scenarios, and further enhances the model’s low-step generation capability through score distillation. Experiments and user studies show that Hyper-SD achieves state-of-the-art performance in models such as SDXL and SD1.5.

Magic Dock Community Model Link:

https://modelscope.cn/models/bytedance/hyper-sd

HuggingFace model link:

https://huggingface.co/ByteDance/Hyper-SD

Project Link:

https://hyper-sd.github.io/

Model Experience

Click the link at the end of the article and you can experience it on the mini program side as well

Diffusers model inference

Diffusers inference example code (single A100 card):

import torch
from diffusers import FluxPipeline
from modelscope import snapshot_download
from modelscope.hub.file_download import model_file_download
base_model_id = snapshot_download("AI-ModelScope/FLUX.1-dev")
repo_name = model_file_download(model_id='ByteDance/Hyper-SD',file_path='Hyper-FLUX.1-dev-8steps-lora.safetensors')
# Load model, please fill in your access tokens since FLUX.1-dev repo is a gated model.
pipe = FluxPipeline.from_pretrained(base_model_id)
pipe.load_lora_weights(repo_name)
pipe.fuse_lora(lora_scale=0.125)
pipe.to("cuda", dtype=torch.float16)
image=pipe(prompt="""a photo of a cat, hold a sign 'I love Qwen'""", num_inference_steps=8, guidance_scale=3.5).images[0]
image.save("output.png")

Raw image speed (3 seconds):

Video memory usage (40G):

ComfyUI Workflow Experience

The memory usage for inference with diffusers is relatively high. Below, we demonstrate using the ComfyUI on Maodai’s free computing power (22G) to achieve real-time generation of images and multi-Lora fusion with the Flux fp8 model + Hyper-SD.

Environment Configuration and Installation:

1. Python version 3.10 and above
2. It is recommended to use PyTorch version 2.3 and above.

3. It is recommended to use CUDA version 12.1 or above.

   This article experiences free GPU computing power provided by the Modao community

Download and deploy ComfyUI

Clone the code and install the relevant dependencies. The dependency links are as follows:

-https://github.com/comfyanonymous/ComfyUI

https://github.com/ltdrdata/ComfyUI-Manager

-https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes.git

# #@title Environment Setup

from pathlib import Path

OPTIONS = {}
UPDATE_COMFY_UI = True  #@param {type:"boolean"}
INSTALL_COMFYUI_MANAGER = True  #@param {type:"boolean"}
INSTALL_CUSTOM_NODES_DEPENDENCIES = True  #@param {type:"boolean"}
INSTALL_ComfyUI_Comfyroll_CustomNodes = True #@param {type:"boolean"}
INSTALL_x_flux_comfyui = True  #@param {type:"boolean"}
OPTIONS['UPDATE_COMFY_UI'] = UPDATE_COMFY_UI
OPTIONS['INSTALL_COMFYUI_MANAGER'] = INSTALL_COMFYUI_MANAGER
OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES'] = INSTALL_CUSTOM_NODES_DEPENDENCIES
OPTIONS['INSTALL_ComfyUI_Comfyroll_CustomNodes'] = INSTALL_ComfyUI_Comfyroll_CustomNodes
OPTIONS['INSTALL_x_flux_comfyui'] = INSTALL_x_flux_comfyui

current_dir = !pwd
WORKSPACE = f"{current_dir[0]}/ComfyUI"



%cd /mnt/workspace/

![ ! -d $WORKSPACE ] && echo -= Initial setup ComfyUI =- && git clone https://github.com/comfyanonymous/ComfyUI
%cd $WORKSPACE

if OPTIONS['UPDATE_COMFY_UI']:
  !echo "-= Updating ComfyUI =-"
  !git pull


if OPTIONS['INSTALL_COMFYUI_MANAGER']:
  %cd custom_nodes
  ![ ! -d ComfyUI-Manager ] && echo -= Initial setup ComfyUI-Manager =- && git clone https://github.com/ltdrdata/ComfyUI-Manager
  %cd ComfyUI-Manager
  !git pull

if OPTIONS['INSTALL_ComfyUI_Comfyroll_CustomNodes']:
  %cd ..
  !echo -= Initial setup ComfyUI_Comfyroll_CustomNodes =- && git clone https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes.git

if OPTIONS['INSTALL_x_flux_comfyui']:
  !echo -= Initial setup x-flux-comfyui =- && git clone https://github.com/XLabs-AI/x-flux-comfyui.git
  
if OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES']:
  !pwd
  !echo "-= Install custom nodes dependencies =-"
  ![ -f "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py" ] && python "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py"

!pip install spandrel

Download the models (including the Flux.1 base model fp8 version, encoder model, vae model, Lora models such as Hyper-FLUX.1-dev-8steps-lora, etc.), and store them in the relevant subdirectories under the models directory. Friends can choose the models they wish to use and download them.

#@markdown ###Download standard resources
%cd /mnt/workspace/ComfyUI
### FLUX1-DEV
# !modelscope download --model=AI-ModelScope/FLUX.1-dev --local_dir ./models/unet/ flux1-dev.safetensors
!modelscope download --model=AI-ModelScope/flux-fp8 --local_dir ./models/unet/ flux1-dev-fp8.safetensors

### clip
!modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ clip_l.safetensors
!modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ t5xxl_fp8_e4m3fn.safetensors

### vae
!modelscope download --model=AI-ModelScope/FLUX.1-dev --local_dir ./models/vae/ ae.safetensors


### lora
#!modelscope download --model=FluxLora/flux-koda --local_dir ./models/loras/ araminta_k_flux_koda.safetensors
!modelscope download --model=FluxLora/Black-Myth-Wukong-FLUX-LoRA --local_dir ./models/loras/ pytorch_lora_weights.safetensors
!modelscope download --model=FluxLora/FLUX1_wukong_lora --local_dir ./models/loras/ FLUX1_wukong_lora.safetensors
!modelscope download --model=ByteDance/Hyper-SD --local_dir ./models/loras/ Hyper-FLUX.1-dev-8steps-lora.safetensors

Running ComfyUI with cloudflared

!wget "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/cloudflared-linux-amd64.deb"
!dpkg -i cloudflared-linux-amd64.deb

%cd /mnt/workspace/ComfyUI
import subprocess
import threading
import time
import socket
import urllib.request

def iframe_thread(port):
  while True:
      time.sleep(0.5)
      sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      result = sock.connect_ex(('127.0.0.1', port))
      if result == 0:
        break
      sock.close()
  print("\nComfyUI finished loading, trying to launch cloudflared (if it gets stuck here cloudflared is having issues)\n")

  p = subprocess.Popen(["cloudflared", "tunnel", "--url", "http://127.0.0.1:{}".format(port)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  for line in p.stderr:
    l = line.decode()
    if "trycloudflare.com " in l:
      print("This is the URL to access ComfyUI:", l[l.find("http"):], end='')
    #print(l, end='')


threading.Thread(target=iframe_thread, daemon=True, args=(8188,)).start()

!python main.py --dont-print-server

Load ComfyUI flowchart link

Hyper-SD single lora accelerated raw image:

https://modelscope.oss-cn-beijing.aliyuncs.com/resource/workflow-flux-hypersd-816-steps.json

Hyper-SD Multi-Lora Fusion Acceleration Live Image:

https://modelscope.oss-cn-beijing.aliyuncs.com/resource/FLUX.1-multilora.json

Thank you to the community developers for sharing the flowchart

Hyper-SD single lora accelerated raw image:

https://openart.ai/workflows/bulldog_fruitful_46/flux-hypersd-816-steps

Hyper-SD Multi-Lora Fusion Accelerated Raw Image:

https://www.liblib.art/modelinfo/ff3f9fe8c41e4c8a9780c728a85f1643

The single Lora flowchart is as follows:

The multi-Lora flowchart is as follows:

Video memory usage (17G):

Note that after uploading the image, it is necessary to check whether the model file in the image matches the name of the model stored for download. You can simply click to associate it with the stored model file name like ckpt_name.

Click the link to jump to the model~

https://modelscope.cn/models/bytedance/hyper-sd